Realistic cloth augmentation in single view video under occlusions

Augmenting cloth in real video is a challenging task because cloth performs complex motions and deformations and produces complex shading on the surface. Therefore, for a realistic augmentation of cloth, parameters describing both deformation as well as shading properties are needed. Furthermore, objects occluding the real surface have to be taken into account as on the one hand they affect the parameter estimation and on the other hand should also occlude the virtually textured surface. This is especially challenging in monocular image sequences where a 3-dimensional reconstruction of complex surfaces is difficult to achieve. In this paper, we present a method for cloth retexturing in monocular image sequences under external occlusions without a reconstruction of the 3-dimensional geometry. We exploit direct image information and simultaneously estimate deformation and photometric parameters using a robust estimator which detects occluded pixels as outliers. Additionally, we exploit the estimated parameters to establish an occlusion map from local statistical color models of texture surface patches that are established during tracking. With this information we can produce convincing augmented results.